Automatic coding of dialogue acts 5 in collaboration protocols 6

11 Abstract Although protocol analysis can be an important tool for researchers to investigate 12 the process of collaboration and communication, the use of this method of analysis can be 13 time consuming. Hence, an automatic coding procedure for coding dialogue acts was 14 developed. This procedure helps to determine the communicative function of messages in 15 online discussions by recognizing discourse markers and cue phrases in the utterances. Five 16 main communicative functions are distinguished: argumentative, responsive, informative, 17 elicitative, and imperative. A total of 29 different dialogue acts are specified and recognized 18 automatically in collaboration protocols. The reliability of the automatic coding procedure 19 was determined by comparing automatically coded dialogue acts to hand-coded dialogue 20 acts by a human rater. The validity of the automatic coding procedure was examined using 21 three different types of analyses. First, an examination of group differences was used 22 (dialogue acts used by female versus male students). Ideally, the coding procedure should 23 be able to distinguish between groups who are likely to communicate differently. Second, to 24 examine the validity of the automatic coding procedure through examination of 25 experimental intervention, the results of the automatic coding procedure of students, with 26 access to a tool that visualizes the degree of participation of each student, were compared to 27 students who did not have access to this tool. Finally, the validity of the automatic coding 28 procedure of dialogue acts was examined using correlation analyses. Results of the 29 automatic coding procedure of dialogue acts of utterances (form) were related to results of a 30 manual coding procedure of the collaborative activities to which the utterances refer 31 (content). The analyses presented in this paper indicate promising results concerning the 32 reliability and validity of the automatic coding procedure for dialogue acts. However, 33 limitations of the procedure were also found and discussed.